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This collection of conference proceedings is based on papers which were 
presented at the Forum on Impact Studies organised by the Education 
Division of the Department of International Development between 24 and 
25 September 1998. The forum brought together participants from a wide 
range of projects in India, South Africa, Morocco, Eritrea, Columbia, Mexico 
and Romania, as well as a number of specialists with experience in 
participatory approaches to project evaluation. Participants brought to the 
conference a profound theoretical understanding of participatory 
approaches to assessing impact - as well as the richness of their personal 
experience gained from many years of practice. DFID also welcomed 
representatives from the British Council, CfBT, Link Africa and Book Aid 
International, who provided a useful stakeholder dimension from a 
management and/or professional standpoint. The majority of the 
participants had been associated with Project Impact Assessment, which 
had been supported by DFID during the 1990s. 

The Forum used the following definition of impact, taken from DFID’s 
Glossary of Aid Terms (1998), as its point of departure: 

Impacts (also referred to as effects) may be planned 
or unplanned; positive or negative; achieved 
immediately or only after some time; and sustainable 
or unsustainable. ... Impacts may be observable/ 
measurable during implementation, at project 
completion, or only some time after the project has 
ended. Different impacts may be experienced by 
different stakeholders. 
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This definition set the theme for the conference and provided useful guidelines 
for focussing dialogue about impact assessment. 

The original programme was slightly modified after a planning meeting which 
some participants attended in March. Although there was broad consensus at 
the Forum about the advantages of following a participatory approach for 
assessing project or programme impact, it was also apparent that some of the 
defining characteristics of this kind of evaluation strategy raised questions that 
needed to be explored and answered by the main stakeholders in the exercise 
from the very outset. The following extracts highlight the issues that emerged 
most prominently, and focus on those factors which the Education Division 
needs to take into account when applying participatory approaches to the 
future evaluation of the impact of education projects or programmes. 

Although participatory approaches to impact assessment were becoming 
increasingly common during the late 1980s, John Shotton pointed out in his 
keynote paper that the World Education for All Conference at Jomtien in 
1990 marked a watershed in terms of local ownership and control in basic 
education programmes and that it included a substantial critique of donor- and 
lender-directed approaches to evaluation. The subsequent decisive shift 
towards wider stakeholder involvement, not only in project implementation, but 
also in project design, monitoring and evaluation, substantially altered the 
imbalance implicit in the donor/beneficiary or donor/recipient relationship. The 
implicit context of the DFID Forum was therefore the learning process which 
has impacted on all funding agencies in the nineties - an era described by 
Kenneth King (1991) as “the Post Jomtien curriculum” - and many contributors 
referred in one way or another to this transformation of the discourse. 

The following themes were used to systematise the papers collected in this 
volume: 

• What is an impact study and how should we do it? 

• The role of base line studies in impact assessment 

• The relationship between national and external researchers 

• Training teachers as researchers 

• Topicality vs. sustainability 

• Impact assessment and sustainability 

• Anticipated and unanticipated project benefits 

The papers in this collection address these issues. The collection is divided 
into thematic sections, and each section deals with a particular theme. The 
papers, as one might expect, vary in style according to the perspective, 
concerns and experience of each author. The variety thus reflected emanates 
from a rich diversity of experience and knowledge. While some participants 
spoke about projects and programmes from the insider’s point of view, others 
were able to complement these contributions with the point of view and 
concerns of those who engage projects and programmes as outsiders. Where 
possible, we have tried to cluster papers which deal with the same projects or 
similar issues. While the collection touches most of the current issues that may 
arise in the conduct of impact studies, it does not pretend to offer a blueprint or 
recipe for undertaking impact assessment. It’s purpose is rather to contribute 
to the development of a participatory framework for impact assessment 
through an exploration of current problems, solutions and suggestions for 
future policy and practice. 



18 



While the final chapter attempts to draw together the conclusions drawn by 
the Education Division from the Forum debate, it should not be understood 
as (necessarily) reflecting the views of DFID as a whole. 




What Is an iw^paci study and how should' we do If? 



I 



0 



I 



The collection opens with a paper entitled Participatory impact 
assessment by John Shotton. His paper sets the scene by 
answering the question What is an impact study? He then elucidates 
what is meant by a participatory study and in so doing, he contrasts 
traditional approaches to the assessment of impact by way of 
participatory approaches - approaches that have come of age in the 
nineties. Shotton’s paper describes the changes in the theory and 
practice that are evidenced in the field of project impact assessment 
since the Jomtien Conference - a period which he terms “the post- 
Jomtien era”. Shotton refers to some of the most far-reaching 
methodological innovations of the decade, such as the 
implementation of the ideals of local ownership and the development 
of local capacity through enabling participants to learn on the job. 
These changes necessitate a shift towards participatory methods, a 
shift which has radical epistemological implications for the 
assessment of projects. 




Veronica McKay’s paper elaborates on the epistemology underlying 
participatory approaches. She argues that participatory approaches 
have an educative function which cannot be replicated by traditional 
approaches to impact assessment. When located within an action 
research framework, she adds, participatory approaches offer 
substantial benefits to all evaluators and impact crucially on the 
development of local capacity. (She points out that participatory 
approaches are contingent on the discursive nature of knowledge - 
an assumption that underlies collaborative enquiry). McKay’s paper, 
however, also draws attention to difficulties that participatory 
researchers may encounter when attempting participatory research in 
disadvantaged communities. She mentions - in particular - difficulties 
associated with enabling and motivating grassroot participation. 



11 




Sasidhara Rao too stresses the importance of the evaluation process 
being informed by a participatory philosophy. He refers to the way in 
which the Andhra Pradesh District Primary Education Programme 
was evaluated and argues that that methods and the instruments that 
were used for this assessment contributed in different ways to 
engaging participation at different levels and at different stages of the 
research enterprise. He indicates that participatory approaches 
encourage participants to reflect on their own contexts - and that this 
informs practice. Participatory approaches, used in this way, also 
mediate a formative function. He indicates moreover that the nature 
of participatory assessments helps to ensure that data - even 
quantitative data - are interpreted in context. 



ERJC 
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N V Varghese considers the distinction between an impact study and 
an evaluation. He outlines four main features which differentiate 
evaluations from impact studies. The distinction, he argues, has 
implications for who conducts the assessment and whose interests 
are most likely to be served. He indicates that while communities 
have more interest in the assessment of impact, the interests of 
funders lie in the evaluation of projects. These observations resonate 
with the critique of the donor-lead forms of assessment presented by 
Shotton in the first paper. Varghese makes a strong case for the use 
of participatory approaches by referring to the human condition. He 
argues that human volition compels researchers to use approaches 
that account for human experience and that participatory approaches, 
by their nature, take this into account. 

While the Forum agreed broadly about the virtues of participatory 
research, it was nevertheless apparent that this form of evaluation is 
more complex than it might appear at face value. The subsequent 
sections in this collection explore these complexities. 




The role of baseline sidles In the assessment of Impaet 




This section opens with Carol Moloney’s paper entitled School- 
focused baseline assessments as a catalyst for change. Moloney too 
draws on the participatory paradigm and argues that when applied to 
baseline assessments, the investigation fulfils an important 
developmental function for teachers. She refers in her paper to her 
experience in training South African teachers to do baseline studies 
and she argues that this exercise achieved benefits which go beyond 
the mere acquisition of data. Indeed, she indicates, teachers acquired 
a modus operandi for doing collaborative baseline studies while 
simultaneously coming to grips with many of the new demands that 
are being made on teachers. 



0 ) 0 ) 



Samir Guha Roy’s paper offers a general framework for evaluating 
educational programmes. He indicates that although participatory 
approaches to assessment have gained ground over the past few 
years, he believes that there is still a need for scientific approaches to 
assessing impact. He argues that scientific approaches may be 
usefully employed in assessing those issues of impact which are 
difficult to assess since they may result from factors that are 
extraneous to programme activities. While firm baseline data and 
regular post-baseline checks could offer a way of overcoming this 
problem, Roy argues that, in the domain of impact assessment, there 
is a growing interest in scientific assessment as a way of statistically 
accounting for such changes. 




Tony Luxon’s paper is the final paper in this section. Luxon’s paper, 
QC g) entitled Issues to consider when planning a baseline study, like 

Moloney’s, deals with the importance of the baseline investigation as 
a way of ensuring adequate benchmarking. Luxon refers to the 
evolution of the philosophy informing the methodology used for 
baseline studies for projects in all types of social and educational 
contexts. It was this paradigm shift, coupled with his own 
experiences in the field of impact assessment, that prompted him to 
compile what he considers to be the twelve essential issues that 
need to be considered each time a baseline activity is contemplated. 
These twelve points (which affect all baseline studies) give rise to 
suggestions about the design of the research approach, the 
selection of members for the project implementation team and the 
dissemination of the findings of the baseline study to various 

stakeholders. 

The concern with the stakeholder is one that all contributors to the 
collection emphasised. The next section is devoted entirely to 
exploring issues related to stakeholder perspectives. 




Who are stakeholders? What are their roles? How do we deal with 
multiple stakeholders with divergent interests? These are the kinds of 
questions that this section on stakeholder perspectives attempts to 
address. 




This section starts with Dermot Murphy and Pauline Rea-Dickins’s 

consideration of stakeholder perspectives. In this paper, they 
emphasise how important it is for evaluators to identify stakeholder 
groupings if they want to make effective use of participatory 
evaluations in educational development projects. They suggest that 
most definitions overlook dimensions of power and power 
differentials, and, as such, are inadequate. This claim underlies their 
view that what is needed is a framework which is more robust- a 
framework that takes issues of power into consideration. Murphy 
and Rea-Dickins present an outline of such a framework - for which 
they find support in their research - and they present an elucidation 
of the implications their findings might have for the practice of 
participatory research. In their discussion of stakeholders, they 
inevitably mention the role of the external consultant. In this regard, 
they coin a term FIFO consultants to refer to those consultants who 
rapidly fly-in/fly-out. They argue, in line with the emphasis that they 
place on participation, that a more sustained consultancy role needs 
to be factored into project designs. Their view of the role of the FIFO 
consultant has repercussions for insider-outsider involvement in 
participatory studies - an issue which is dealt with in detail in the 
fourth section of this collection. 
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Clara Ines Rubiano and Dermot Murphy in their paper entitled 
Considering the audience - an important phase in project 
evaluations, emphasise how important it is for evaluators to give 
consideration to those audiences for whom the evaluation is intended. 
They interrogate complexities associated with notions of audience, 
and refer to the differing interests, differing statuses as well as the 
differing power relations that are inherent to the concept of audience. 
They argue that the identification of, and consideration for, the 
audience/s is central to notions of the practical utility of the 
recommendations of an evaluation and to the compilation of 
evaluation reports. The authors draw on critical incidences pertaining 
to the audience/s, which manifest in the evaluation of the Colombian 
Framework for English Project. 




In this paper Coco Brenes and Tony Luxon consider the variety of 
audiences that are implied by multi-partnered projects and the 
implications of the varied audiences for the dissemination of the 
project report, and for the mode of reporting. The paper considers the 
complexities implied by the reporting process, and in particular 
addresses issues such as: Who writes the a report? Who reads it? In 
what language is it produced? and How is it disseminated? Each of 
these questions is addressed against the backdrop of the ODA ELT 
Project in Nicaragua. 



SECTION 4: 

The relationships between national and external researchers 
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Tesfai Bariaghaber’s Note on a participatory impact study, which is 
the result of his personal involvement in Eritrea, explores the relations 
between national and external researchers. He begins his paper by 
stating that the assessment study was characterised by both highs 
and lows. Hence, while his paper acknowledges the many 
advantages of participatory impact assessment, he nevertheless 
refers also to some of the lows. The lows as described by him might 
be likened to the effects of FIFO (defined earlier in paper 3.1). For 
him, the lows are primarily a result of the geographic divide between 
the local and external evaluation team. He contends that in the case 
of the Eritrean assessment, the external evaluators moved out of the 
project too soon to allow them to make a meaningful contribution to 
the development of local capacity. Their early departure had 
ramifications for their stake in the ownership of the assessment 
project. The departure of the external consultants prevented the local 
team from being able to contribute further ideas or recommendations 
to the research report. He nevertheless concludes by indicating that 
collaborative research is beneficial to both internal and external 
researchers. 
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Harvey Smith and Paul Bennell also, as their paper title indicates, 
draw out the complexities associated with the Relationships between 
national researchers and external researchers. Their paper is based 
on their personal experience of impact studies in which they were 
engaged in Angola and Eritrea. As with paper 4.1 , this paper 
describes a series of conceptual and practical project issues that 
impact on the relationship between internal and external researchers 
- in particular those that give rise to ownership-type problems. They 
argue that there is a need to achieve the “correct balance between 
local and external ownership”. They believe that this correct balance 
can only be achieved by a research design that ensures that the 
study meets local needs, and that the external funding agencies are 
empowered to rate a project’s achievements. In their conclusion they 
ponder the kinds of compromise that might be possible. 

Mohammed Melouk, in his paper entitled The role of an insider/ 
outsider in impact assessments, also explores complexities pertaining 
to the relationship between internal and external researchers. He 
bases his discussion on his experience of the Moroccan ELT 
project, and refers to some of the many complexities implied by the 
researcher’s roles. He argues against the imposition of investigations 
or project designs without these being grounded in a solid local 
perspective. In highlighting the need for insiders to participate in the 
assessment of impact, he coins the phrase insider/outsider, which 
refers to those locals who are outsiders to the project - but who are 
insiders to its situational context. He outlines several good reasons 
why insiders/outsiders should be included in impact assessment 
teams - not only because they are communication facilitators but also 
because of their ability to mediate data and thereby contribute to 
insightful and contextually appropriate conclusions. In this sense he 
echoes Rao’s sentiments in paper 1 .3. 



Dave Allan’s paper is the final paper in this section. His paper, 
entitled Impact assessment in educational projects: some 
perspectives on the ‘insider-outsider’ debate, also stresses the 
importance of a consideration of the roles of insiders-outsiders in 
project assessment. These roles, he argues, have implications for 
who does the evaluating and who decides whether the outcomes are 
judged as either successful or not. In order to situate his own position 
in this debate, he draws on four case studies of evaluations 
undertaken in Bangladesh, Estonia and Morocco. These evaluations 
reflect a variety of permutations on a continuum from, on the one 
hand, being a single outsider researcher to, on the other hand, 
working as an insider with a range of insider-stakeholders. Allan 
makes various recommendations for good practice on the basis of his 
observations and critique. 
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Training teachers as researchers 



In this section, both Peacock’s and Sekgobela’s papers focus on training 
teachers to do assessments. They claim a wide range of benefits as a 
result of this training. Earlier papers also made some reference to teachers 
doing research: the papers of Moloney and McKay, for example, draw 
attention to the many benefits of this form of training. 



Alan Peacock, in his paper entitled Helping teachers to develop 
competence criteria for evaluating their professional development, 
discusses interventions in South Africa and Sri Lanka which were 
intended to help teachers to develop competence criteria for 
evaluating their professional development. He elucidates various 
stages of the process which enabled teachers to evaluate their own 
performance by developing criteria for assessing competence in 
teaching. Teachers may apply the criteria, which they have 
generated, as part of a collective enterprise in their classroom 
situations. He argues that the reflection and thinking underlying this 
approach enabled teachers to become aware of the need to establish 
levels of achievement in any given skill area. In practice this meant 
that teachers are given the responsibility to develop their own 
competence. This obviously has a number of positive spin-offs for 
their teaching practice. 

[cF 1 0) Elijah Sekgobela also trains teachers to do research. In his paper, 
entitled Combining the teaching of research methods with the 
assessment of project impact, Sekgobela describes how, while using 
the University of South Africa’s (Unisa) training course for adult 
educators to teach research skills which are needed by students for 
the fulfilment of curriculum requirements for trainee educators, he 
simultaneously uses his teaching opportunities to conduct impact 
assessment. In this paper, he describes the process which required 
students to participate in all spheres of the research process - from 
the initial conceptualisation of the research to the final stage of 
recording of data. This paper discusses the process and benefits 
derived from teachers’ undertaking an evaluation of their own 
contexts. 




SECTION 6: 

Topicality vs. sustainability 





Jeff Samuelson and Sara Harrity consider the debate which has 
arisen from attempts to answer the questions What outcomes are we 
looking for in terms of impact and what are the implications for the 
approach that we may adopt? In answering these questions they 
draw on two projects with which Book Aid International is associated. 
These projects, they argue, focus more on outputs than on issues of 
sustainability. They argue that, by their very nature, these outputs 
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may be described as addressing questions of topicality rather than 
sustainability, the necessity for maintaining an accurate focus is made 
more complex by the requirement that evaluators determine the extent to 
which a particular project’s intervention (as opposed to any number of 
external influences) has brought about whatever changes may have been 
observed. If the assessment is to determine what impact has been a 
direct result of the intervention and what impact should be attributed to 
extraneous factors, it must consider a number of other features such as, 
for example, the political, social and economic context in which the 
project has been operating. 

/T* <Z7) Cleaver Ota’s paper echoes the concern expressed by Samuelson and 
Harrity - that it is essential to address features of the context in our 
endeavours to attain a prognosis for project sustainability. His paper 
outlines the approach employed in the determining the outcomes of the 
South African Book Aid Project (SABAP) and certain concerns pertaining 
to project sustainability. While he concludes that the project had achieved 
the outcomes defined in the project document, he points to extraneous 
factors which impinge on these achievements. Accordingly, he asserts 
that it is not possible to assess impact, or to speculate on sustainability 
without locating the project within its socio-economic and political context. 
To do so, would be tantamount to decontextualising the possibilities for 
delivery. This, he argues, is because there are a number of extraneous 
factors which impinge on the actual implementation and which have a 
bearing on the potential for sustaining the project. With regard to the 
SABAP project, he identifies two such features: namely the role of 
government in financing the post-donor phase of the project, and the 
complex relations implied by collaborative multi-partnered 
implementation. 



SECTION ?: 

Iwip^ict ^ssessitient and sustainability 



This section focuses on the relationship between the assessment of impact and 
project sustainability. The papers take as their point of departure, the way in 
which the form of assessment could contribute to the enhancement of project 
goals, and to the capacitation of local players. These discussions are 
juxtaposed with the approach employed to assess the impact made by Mexican 
Advanced Professionalisation Project. 




This section opens with the speech delivered by Carew Treffgarne on 
behalf of the Latin America, Caribbean and Atlantic Department (LACAD), 
DFID at various regional conferences on the impact of the 
professionalisation of the english teacher training in Mexico. Her talk 
outlines the rationale underlying the design of the model for assessment, 
and as such, provides an informative backdrop for the subsequent papers 
in this section. Her paper is intended to situate the collection of papers in 
this section - all of which are based on the evaluation of the Mexican 
Advanced Professionalisation Programme. Treffgarne indicates that 
the decision to use a participatory approach to evaluating project impact, 
was based on the understanding that it was not possible to do justice to a 
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programme (of the scope and scale of MAPP) by utilising traditional 
approaches to evaluation. Her talk outlines some of the benefits of 
participatory assessment - in particular with regard to furthering the 
achievement of project outcomes, and to enhancing possibilities of 
sustaining project benefits. Her paper suggests that the approach 
employed, would be of direct benefit to participating universities, and 
more broadly, to the sector. Her paper provides an informative 
backdrop for the subsequent discussions of Morrow, Basich and 
Rodriguez. 

Following Treffgarne, Keith Morrow’s paper is concerned with the 
extent to which projects are able to sustain their impact after the 
project is concluded. His paper focuses on the assessment of impact 
on participants in the Mexican Advanced Professionalisation 
Project, a project intended to upgrade the professional qualifications 
of teachers working in language departments. Morrow describes the 
approach used to gain a sense of the impact made on teachers - an 
approach to the assessment of impact which doubled up, in formative 
fashion, as a component in building of institutional capacity. In this 
sense, Morrow views participatory research approaches as being 
essential for sustainability because they provide participants in 
particular with an opportunity to undertake a qualitative and 
quantitative assessment of impact. He suggests that this is one way 
to enhance the professionalism of those involved. He also indicates 
that the process of evaluation, constructed along similar lines, could 
contribute to the aims of the project - while at the same time 
contributing to sustainability. 

Kora BasichPeralta’s paper also elucidates aspects of the Mexican 
Advanced Professionalisation Project’s assessment of impact. She 
outlines the research approach employed by her university in 
assessing impact. She mentions that once the assessment had 
begun, the research team was amazed to discover the achievement 
of outcomes which were not initially anticipated. In particular, she 
refers to the achievement of sector-wide, as well as institutional and 
policy outcomes. Basich, like Morrow (see the previous paper) 
indicates that the process of evaluation - especially the reflective 
component - achieved more than just the necessary required data. It 
also, she indicates, contributed to the enhancement of project goals 
in terms of qualitative improvement of English teacher training. 

In this paper, Jorge Anguilar Rodriguez describes the method of 
assessment used in the Mexican Advanced Professionalisation 
Scheme (MAPS). He indicates that although the research design 
utilised in this project was similar to standard research designs used 
elsewhere, the emphasis in this kind of assessment is different. The 
emphasis in the research design was directed at uncovering inter alia 
the unanticipated outcomes - and these, once discerned, played a 
significant role in ensuring project sustainability through their being 
posed as benchmarks for the continuation of the MAPS programme 
and for new projects which might be started. 
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Mfanwenkosi Malaza, like Samuelson and Harrity, also suggests that 
if the assessment is to determine what impact has been a direct result 
of the intervention and what impact should be attributed to extraneous 
factors, it must consider a number of other features such as, for 
example, the political, social and economic context in which the 
project has been operating. He also examines the anticipated and 
unanticipated benefits of a project. His paper focuses on the types of 
impact made by the Mpumalanga Primary Schools Initiative. He 
argues that the determination of a project’s benefits is more complex 
than it may appear to be at face value and that it is necessary to 
distinguish between anticipated and unanticipated outcomes. He 
argues that every project has degrees of both intended and 
unintended outcomes - whether they are positive or not - and that 
the impact of the unintended outcomes very often outweighs the 
intended ones from the local people’s point of view. He elaborates on 
the unanticipated outcomes, which were not predicted at the start of 
the intervention but which nevertheless make a significant impact. He 
argues that these need also to be considered when evaluating project 
impact. He, like Samuelson and Harrity, contends that when 
identifying unanticipated benefits, it is necessary to look at the wider 
context of a project’s operational environment. This becomes vital if 
one wishes to guard against attributing effects to the project that are 
merely incidental to it. 



(?) 0 ) Mirela Bardi and Roy Cross also give consideration to the question 
project outcomes. Their paper deals with an assessment of the 
impact of the Project for Special Purpose English in Romania 
(PROSPER). The paper describes how, apart from measuring the 
impact of the project, the evaluation specifically takes into account the 
sectoral impacts that led to the ripple effects of the project. These 
ripple effects mean that the project affects not only the sector, but 
also those institutions which were not participating in PROSPER. 

Bardi and Cross point out that it is necessary to consider the 
consequences of such ripple effects on the sector. 




In her paper, Roopa Joshi attempts to provide a review of a critical 
area of project management in the District Primary Education 
Programme (DPEP), namely that of the assumptions underlying the 
practices and strategies for the assessment of project impact. She 
illuminates three broad issues, which she argues, informed the 
approach: 



Firstly, it was necessary to address the question of how the DPEP 
impact assessment model should be designed. The how, she 
suggests, refers to the design on both a conceptual and operational 
level. 






Secondly, it was necessary to consider the content and range of 
existing DPEP practice as it manifested across the various states and 
at various levels of decision-making. In terms of this, it was pertinent 
to establish how this practice might influence the various stakeholders 
of the project. 

And thirdly, it was necessary to consider what the possible way 
forward might be for DPEP in the arena of assessment research. 

(?) Finally, Carew Treffgarne presents reflections on the contributions 

<2) 0 <— which emanated from the Forum on Impact Studies. In this paper 
she offers her concluding comments. Her reflections include an 
acknowledgement of the value of a participatory approach to impact 
assessment as well as the complexities associated with the process - 
particularly with regard to local ownership, insider/outsiders and, of 
course, the FIFO factor. Her paper draws attention to what DFID 
ought to take into account as it attempts to resolve the problems and 
confront the issues that evaluators have delineated in their papers. 
She analyses the papers in this collection in terms of the same 
thematic categories which have been used to organise this volume. 

Treffgarne recommends that DFID’s Education Division pay serious 
attention to the recommendations about the importance of allocating 
adequate time for assessments, budgetary considerations, and of the 
actual timing of such assessments. Considerations such as these 
need to be factored into project documents and project budgets. Her 
paper makes fundamentally important statements about the 
assessments of baseline studies and project impact - as well as 
about the sustainability of projects. Carew Treffgarne concludes by 
indicating that the Forum on Impact Studies has been instrumental 
in helping Education Advisers in DFID to identify some of the 
important lessons learned from the two-day workshop, issues which 
might constructively inform the future practice of the Department. 



Veronica McKay &. Care w Treffgarne 
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John Shotton 

Centre for Overseas and Developing Education 
Homerton College 
University of Cambridge 




In this paper, John Shotton considers the changes in the theory and 
practice that are evidenced in the field of project impact assessment in the 
post-Jomtien era. He indicates that subsequent to the Jomtien Conference 
in 1990, aid programmes were increasingly characterised by a shift away 
from being funder-driven towards being locally owned and locally driven. 
This paradigm shift has been possible through, inter alia, the development 
of local capacity. This shift, Shotton indicates, has radical epistemological 
implications for the assessment of project impact - an issue that this paper 
interrogates. The author presents a strong case for formative participatory 
impact assessments, which, he argues, contribute to the building of project 
capacity and local ownership. Participatory practice enables participants to 
learn on the job and is more likely to be responsive to local needs than are 
traditional approaches. 

Finally, Shotton demonstrates the shift from traditional forms of 
assessments to participatory assessments by contrasting the assessment 
of projects that might be classified as traditional (pre-Jomtien) with those 
that demonstrate what he considers to be the essential ingredients of 
participatory practice. 
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What is an impact study and how should we do it? 



1 Introduction 



There are three important contexts to this consideration of the nature and 

operation of participatory impact assessment: 

o The first is what King (1991) has called The Post-Jomtien Curriculum. 
This is the learning agenda for the international donor and lending 
agencies laid out by Third World Network at the World Education for 
All Conference (1990) at Jomtien. The agenda centres on issues of 
local ownership and control in basic education aid programmes and 
includes a substantial critique of donor- and lender-directed 
approaches to evaluation in the pre-Jomtien era. 

o The second is the shift in approach of some of the international donor 
and lending agencies in some projects to the Post-Jomtien 
Curriculum. 

o The third is a focus on a sample of basic education programme 
evaluations in an attempt to draw out the essential ingredients of 
participatory impact assessment. The evaluations considered are by 
no means all examples of participatory practice. On the contrary, 

I make comparisons of participatory and more conventional and 



Before we consider participatory approaches to impact assessment, it is 
important to be clear about the nature of impact assessment itself. Impact 
assessment may be distinguished from other types of evaluation by the 
area of the programme on which it focuses. This logic follows the evolution 
of the programme as it unfolds and has been a generally useful paradigm in 
educational evaluation. Rossi and Freeman (1993), for example, distinguish 
between three programme phases which strike me as particularly useful: 

© Conceptualisation and design 

® Monitoring and implementation 

© Assessment of effectiveness 

Each of these phases is compatible with different evaluation strategies: 



At the conceptualisation phase of the programme, a diagnostic evaluation 
procedure may be appropriate as research questions focus on programme 
features such as the programme’s underlying assumptions, its logic, major 
stakeholders, the programme’s objective, and the context in which 
implementation is to occur. Adequate understanding of these issues is 
critical before a programme is designed and started. 




traditional approaches. 



2 is impact ^ssessknent? 



21 Conceptualisation and design 
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Participatory impact assessment 



2.2 Monitoring and implementation 



The second stage, monitoring and implementation, focuses on the 
programme’s operations after the project has started. Here, several types of 
evaluations may be appropriate for a given objective. These are essentially 
formative evaluation approaches and are intended to improve the overall 
operations of the programme. Several different evaluation modes could be 
included in this group including, evaluability assessment, which attempts to 
answer the basic question of whether a programme can be evaluated. 
Perhaps best known though in the process of implementation evaluation is 
what focuses on delivery and assesses the programme’s conformity with its 
basic design. Performance monitoring and implementation indication could 
be included in this group. This type of evaluation periodically reviews the 
short-term outcomes of the programme, along with its quality, to assess the 
degree to which the programme’s activities affect these outcomes. 



2.3 Assessment of effectiveness 

It is in the phase immediately after initial implementation that we find impact 
assessment. Impact assessment gauges the extent to which a programme 
has led to desired changes in the target field and audience. It implies a set 
of programme objectives that can be identified and used as a basis for 
measuring the programme’s impact. Thus the overall goal of an impact 
assessment is to determine if, and the extent to which, a programme has 
met its objectives. In this phase of the programme, distinguishing impact 
from the programme’s outputs and outcomes is often valuable. Outputs 
refer to the immediate consequences of the programme whereas outcomes 
describe the more immediate results. Both outputs and outcomes may be 
intended or unintended, and need to be assessed for their logical 
relationship to final programme objectives. 



24 Formative or swK^ative assessments 

It has often been argued (IDRC 1972) that impact assessment can only be 
summative. However, given the time frame of most basic education aid 
programmes, it is critical that they are formative. As Phile (1994) argues, 
impact assessment and evaluation in general must not simply serve the 
need for the international donor and lending agencies to satisfy their 
respective governments’ treasury departments and banks. On the contrary, 
the priority should be to serve the needs of primary users and it is here that 
a participatory paradigm becomes essential. Though Phile recognises the 
need for the agencies to benefit from evaluation, for him it is only a question 
of pursuing advocacy on the part of primary users as a priority. 

That this is necessary is clear from the principles for the evaluation of 
development assistance set out by OECD (1992:132): 

The main purposes of evaluation are: 

• to improve future aid policy, programmes and projects 
throughfeedback of lessons learned. 

• to provide a basis for accountability, including the provision of 
information to the public. 

”iat is an impact study and how should we do it?. 

eric 
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To this should be added purposes that reflect the conclusions of the 
Jomtien Conference in relation to evaluation, namely that it should assist 
the process of capacity building at the local level and local ownership and 
control in a context of the decentralisation of programme administration. 



3 WhM is participatory impact assessment? 

By participatory impact assessment I am referring to what has been 
described as applied social research that involves trained evaluation 
personnel and practice-based decision makers working in partnership 
(Cousins & Earle 1992). Usually decision-makers are donor or lending 
agency personnel and recipient country administrators with programme 
responsibility, or people with a vital interest in the programme. Participatory 
impact assessment is best suited to formative evaluation exercises that 
seek to understand innovations with the expressed intention of informing 
and improving their implementation. As I indicate later, two projects that fit 
this bill are two of the largest post-Jomtien Education for All (EFA) 
programmes in the world, namely the District Primary Education 
Programme (DPEP) in India and the Effective Schools Through 
Enhanced Educational Management (ESTEEM) programme in 
Bangladesh - the latter two are substantially funded by the Department for 
International Development (DFID). 

In participatory impact assessment, a crucial part of the capacity building 
deemed necessary for evaluation by the Jomtien Conference is to train key 
personnel (project administrative staff) in the technical skills crucial to the 
successful completion of the research exercise. Thereafter, practitioners 
(resource centre staff, teachers and community members, including those 
on school committees, parents and possibly children and other learners) 
can learn on the job with mentoring and workshop input where necessary. 
When this happens, both parties participate crucially in the research 
process. Such learning is an indispensable part of the participatory model 
since the intention is that key administrative personnel develop sufficient 
technical knowledge and research skills to take on the coordinating role in 
continuing and new projects, and that they need to rely on the initial trainer 
for consultation about technical issues and tasks such as statistical 
analysis, instrument modification and technical reporting. Participatory 
impact assessment is likely to be responsive to local needs, while 
maintaining enough technical rigour to satisfy probable critics - thereby 
enhancing use within the local context. 



4 +4ow is participatory (Hip^ict £issessi*»ent different? 

Participatory impact assessment is conceptually distinguishable from other 
types of named collaborative enquiry and evaluation on two important, 
although not independent, dimensions: goals and process. 



Participatory impact assessment 



41 The goals of participatory impact assessment 

In relation to goals, the pre-Jomtien orientations designed by the northern- 
based academic community advocated the simultaneous improvement of 
local practice and the generation of valid social theory (Cochran-Smith & 
Lytle 1993) as in, for example, the so-called state of the art evaluation of 
the elementary education programme in the Philippines in the 1980s. 
Similarly more contemporary practitioner-centred instances of collaborative 
evaluation have expressed as a goal the empowerment of individuals or 
groups, or the rectification of social iniquities. Such a goal is expressed for 
example by the Swedish International Development Agency (SIDA) 
evaluation of the teacher training programmes for primary and secondary 
education in Mozambique and Guinea Bissau in the 1980s and 1990s 
(Carr-Hill 1997). These interests are beyond the scope of participatory 
impact assessment since such interests belong firmly to programme goals 
and programme implementation. I would argue that it is fundamentally 
dishonest to believe that an evaluation process can achieve such ends. 

This would constitute only a reflection of tokenistic commitment to a social 
agenda by non-practitioners more interested in the formulation of grand 
social theories and rhetoric rather than reality: it would be a tantamount to a 
‘deodorant’ that tries to sanitise the inadequacies of overall programme 
direction. 

The approach that I would advocate is not ideologically bound, nor is it 
devoted to the generation of social theory. Rather participatory impact 
assessment has, as its central interest, an intention to enhance the use of 
evaluation data for practical problem solving within the contemporary 
organisational context - an endeavour that will support the overall 
programme goals. Indeed this is the essence of Phile’s argument in relation 
to the post-Jomtien scenario, namely that the driving force for a new 
agenda relies on overall programme definition and orientation and that we 
need to make sure that individual programme components accord with that 
definition and orientation. 



4.2 The process of participatory impact assessment 

The second differing dimension, process-based, takes shape inside 
participatory impact assessment by having administrators and key 
organisational personnel working in partnership with members of the 
community of practice as opposed to other models, such as the benefit 
monitoring model that has served the Nepal Basic Education Programme 
and the Nepal Secondary Education Project through the 1990s, which 
exclude the latter. Whereas administrators, for example, do bring a set of 
technical skills to the evaluation act which are important, practitioners bring 
a thorough knowledge of context and content and the partnership is critical 
for effective participatory impact assessment. The former work as 
coordinators or facilitators of the research project, but fully share control 
and involvement in all phases of the research process with practitioners. 
This thrust is distinguishable both from pre-Jomtien forms of evaluation 
where control of the research process is maintained by the expert evaluator 
or evaluators (Whyte 1991), and from so-called practitioner-centred 
approaches where such control lies completely in the hands of the key 
individuals in the practitioner group (Elliot 1991). 
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4.3 Some references to participatory assessments 



Participatory impact assessment may thus be summarised against what I 
call the pre-Jomtien model, which has often masqueraded as a 
participatory entity: 

o The pre-Jomtien model, the benefit monitoring in Nepal being a 
classic example, attempts to engage many potentially interested 
members of recipient-country administrators in order to create 
support but without yielding any power in the crucial areas of model 
focus and design. The participatory model, envisaged for ESTEEM in 
Bangladesh, will actively involve primary users at all stages of the 
impact assessment process, from focus and design through to 
dissemination of conclusions. 

o The pre-Jomtien model involves programme participants in a 

consultative way to clarify domains and establish the questions for the 
evaluation project. SIDA’s work in Mozambique and Guinea-Bissau 
epitomises this. The participatory model engages the primary users in 
the ‘nuts and bolts’ of focusing the assessment, formulating the 
design, deciding on the methodology and sample, developing the 
instruments for data collection, collecting the data, analysing and 
interpreting the data and reporting the results and making 
recommendations. Possibly the best example of this is the impact 
assessment mechanism that has been developed in Andhra Pradesh, 
India, as part of DPEP. 

o In the pre-Jomtien model, the expert evaluator or evaluators are the 
principal investigators who translate the institutional requirements into 
a study and conduct that study, as in the case of the Philippines 
evaluation already referred to above. In the participatory model, as in 
the case of DPEP Andhra Pradesh, the external consultants help only 
to coordinate the exercise and are responsible for advising about 
technical support, training and quality control. Conducting the study is 
the responsibility of practitioners. 



The underlying justification for a genuinely participatory approach is 
problem solving in professional work, which is closely tied to Schon’s 
(1983) terms: reflection-in-action and reflection-on-action. Through 
participatory impact assessment, recipient country administrators and 
donor and lending agency members may be surprised by what they 
observe and may therefore be moved to rethink their practice. Unlike so- 
called emancipatory forms of action research, that use Participatory Rural 
Appraisal (PRA) for example, the rationale for participatory impact 
assessment resides not in its ability to ensure social justice or somehow to 
level the societal playing field, but in the utilisation of systematically and 
socially constructed knowledge. 
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51 A consideration of the utility of the findings of an evaluation 

I here express my orientation towards evaluation utilisation which suggests that 
under certain conditions, evaluation or applied research data will be used either 
for providing support for discrete decisions in programme constituencies (e.g. 
decisions about programme expansion) or for educating organisation members 
about programme operation and the consequences of programme practices. 
These uses of data are known to be dependent on two main categories of 
factors: 

• features of the evaluation itself, including its timeliness, relevance, quality 
and intelligibility 

• features of the context in which data are expected to be used, such as 
programme implementers needs for information, political climate and 
receptiveness toward systematic enquiry as a mode to understanding 
(Cousins & Leithwood 1986). 

This framework for understanding participatory impact assessment is 
inadequate in at least two respects. 

Firstly, it links the use of data to an undifferentiated individual called the 
decision-maker. To assume that organisational decisions supported by data are 
the product of single individuals processing information and translating it into 
action is, at best, tenuous and probably not representative of decision making in 
most organisations. Rather, decisions made explicitly, or implicitly, are the 
product of some form of collective discourse, deliberation or exchange. As such, 
it is eminently preferable to envision the nature and consequences of 
participatory impact assessment in the context of organisational groups, units, 
subunits and the like. 

Secondly, the evaluation framework may be described as inadequate since it 
fails to recognise the powerful influences of various forms of interaction between 
practice-based and research-based communities. Considerable evidence is 
accumulating to show the benefits of combining the unique sets of skills, 
brought to projects and tasks by both researchers and members of the 
community of practice, regardless of whether or not the tasks are research- 
based. 



Cousins and Earle (1992) have provided a thorough review of a variety of lines 
of research-based evidence in support of the participatory impact assessment 
process. Their findings underscore the importance of social interaction and 
exchange and the need to conceive of organisational processes in collective 
and social terms. They also support the integration of research and practice 
specialisations as a means to stimulating enduring organisational change. An 
appropriate theoretical framework in which to situate participatory impact 
assessment, then, will be one that adheres to such principles. 



Participatory impact assessment, viewed from this perspective, is a strategy or 
intervention that will produce adaptive knowledge to the extent that it monitors 
and provides an opportunity for the interpretation of programme outcomes, and 
generative knowledge such that interpretations lead to enlightenment or the 
development of new insights into programme operations, or effects, or 
especially organisational processes and consequences. 
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6 Conclusion 



Finally, the post-Jomtien changes in the theory and practice of 
project impact assessment have encouraged the shift to 
participatory assessment - an interventionist practice that 
contributes to many dimensions of the project. This is more so 
when participatory assessments are undertaken as formative 
activities. The evaluative assessment can then be regarded as a 
powerful learning system, designed ultimately to foster local applied 
research, and thereby enhance social discourse about relevant 
learning centre-based issues. When applied research tasks are 
carried out by school and district staff, their potential for enhancing 
organisational learning activity will be strengthened and the 
sustainability of the project be enhanced. 
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1.2 



Participatory action research 
as on approach to impact 
assessment 



Veronica McKay 

Institute for Adult Basic Education and Training 
University of South Africa 




In this paper Veronica McKay corroborates John Shotten’s view of 
the post-Jomtien shift towards a participative process for 
researching project impact. In elaborating this point of view, she 
asserts that the participative approach to assessment presupposes 
an epistemological shift from more realist-orientated research 
approaches towards a non-realist approach to assessing impact. 
This view of knowledge, she argues, is diametrically different from 
the positivist belief in an objective reality and knowledge that are 
universally true or false - the epistemological presupposition which 
inspired traditional pre-Jomtien approaches. She argues that a non- 
realist orientation opens the way for multi-vocal discourses, and 
that this is a prerequisite for participation. 

One of the implications of the non-realist epistemology is teachers 
(as active participants) are brought into our endeavours to assess 
project impact. It is only by doing this, she asserts, that we can 
ensure that the assessment of impact will be both formative and 
relevant and educational for teachers at the chalk face. In this 
paper McKay discusses the advantages and problems associated 
with participatory action research (PAR) in general and then 
specifically examines how it may be applied to the assessment of 
impact. She illustrates her points by making reference to the 
application of PAR to the assessment of impact in the Molteno 
Early Literacy and Language Development (MELLD) project in 
Namibia. 
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